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ABSTRACT 

The activities and procedures for Education Service 
Center, Reqion 20, Evaluation Services* time accounting system are 
described. Each individual in the evaluation component at the end of 
the working day records his or her time spent on an activity. This is 
done through the use of the time accounting form which is completed 
and turned in weekly to the person responsible for maintaining the 
records. Activities are recorded in reference to the appropriate 
program, whether chargeable or non-chargeable, as well as the task 
area that effort supports. Printouts by task, project hours, and 
employee and project numbers are obtained monthly. These printouts, 
shown in appendix B, are basically a management tool for control of 
the total evaluation effort on the project. Totals are maintained and 
recorded monthly, so that the data on the hourly effort remaining in 
each grant of the pVogram are readily available. The system can be 
replicated by any school district that uses multiple funding sources 
in supporting personnel. Such time and effort record keeping systems 
are now required ifhen a person is fundevi from several federal or 
state funding sources. (Author/PN) 
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ABSTRACT 

How much time is needed to promote use of evaluations? This 
research suggests answers. The setting is an evaluation office within 
a regional education service agency in the state of Texas. Data is 
based on time and effort records from over forty evaluations. Evalua- 
tions examine both district and center sponsored programs. 

What was found? Eleven percent of an evaluator^s time was budgeted 
for use. More time was spent on use for Center-based special education 
than curricular programs. Internal Center -based evaluators tended to 
dedicate more time to use than the external District-based evaluator. 
Approximately equal amounts of time are spent by the internal and 
external evaluators if time spqnt on technical assistance activity is 
included with other use records. • 



HOW MUCH time; is Ri^QUIRED 
I'OR USING l";VALl)AT10N RliSUI/rS? 

Most evaluation?; of social progrmns do not make a difference. 
(David, 1978; Alkin, and Daillak, 1979). Findings are seen as irrele- 
vant. Seldom are decisions based on evaluative information. If the 
profession of evaluation were judged by use of results, it would receive 
failing marks. Nevertheless, in some settings, evaluations are used for 
program improvement. IVhat are the salient characteristics of successful 
settings? The often troublesome political context, for exainple, is com- 
bated by strong institutional support for evaluation. (Davis and Salasin, 
1973) . The evaluation office should be independent of programs it evalu- 
ates (Anderson and Ball, 1978). The personal factor must be present 
someone who really cares about the program and the evaluation must be 
identified (Patton, 1978). Program personnel should have good management 
skills (Cox, 1977) and fully understand program intent; evaluation staff 
should comprehend program activity and possess good communication and 
interpersonal skills (Holley, 1979). 

The process of using evaluative information is characterized by close 
working relationships between evaluation and program personnel. Unfortu- 
nately, little has been written about the nature of this process. Little, 
if any, research has been carried out on how much evaluation time should 
be allocated for use of information. Being able to estimate this amount 
of time should represent an important starting point in carrying out evalu- 
ations. Ivnowing what varial)les influence this allocation process (e.g., 
content area, evaluator role external or internal, newness .of program, 
etc.) could dramatically affect better use of evaluations. 



The purpose of this research is to document how much time is being 
dedicated for use of evaluation findings. The setting for the study is 
im evaluation orrice located in a regional education service agency. The 
office has good use of evaluations when compared to other offices."^ It 
receives strong administrative support and evaluation personnel are in- 
dependent of the programs they evaluate. 

THE SETTING (ESC- 20) 

Education Service Center, Region 20 (ESC-20) is one of twenty regional 
education service agencies in the state of Texas. ESC-20 serves fifty 
school districts in a fourteen coUnty area. Services provided are evalu- 
ated annually l:>y an evaluation office consisting of eleven full-time and 
nine part-time staff. Since evaluation services are being funded from 
more than one source, it is necessary that records be maintained to show 
how much effort, in terms of evaluation time, is spend on each account. 
The amount requested for evaluation is based on time and effort records 
reflecting the amount of work necessary to conduct evaluation of programs 
of similar scope and type. In actual practice, each individual in the 
office, except clerical personnel, at the end of the working day, records 
the amount of time spent on each activity for each program. A detailed 
description of the system is given by Appendix 1. 

METHOD 

As noted earlier, this research examines the amount of evaluator 
time dedicated to the use of information. How is this time captured? 



The cv'iiluator codes instances involving use. These include: 
. Developing or finalizing evaluation plans 
. Providing technical consultation 

. Verbally conveying findings or results of evaluation 

. Formulating recommendations 

. Consultation Involving using evaluation data 

This research spans records kept for 17 months, from September 1980 
to January 1982. For this period of time, over 40 evaluations are 
represented. They are distributed among three areas: Center-sponsored 
curriculum programs. Center -sponsored special education programs and 
District programs. Many of the District programs involved bilingual edu- 
cation. The evaluation staff has remained essentially the same for the 
period under study. There were 14 evaluators in 1980-81 and 11 evaluator 
in 1981-82. Evaluators are generally assigned to both Center-based and 
District work. 

" I'or this research, the evaluator is an internal evaluator for Center 
sponsored programs. The evaluator and program staff belong to the same 
organization. For Districts, however, the evaluator serves an external 
role. The evaluator is part of the staff cf ESC 20. Program personnel 
work for another organization, namely, the school district. 



RESEARCH QUESTIONS 



T. How much time should be set aside to promote use of evaluative 
iafq^nnat:Von? 

IT. Are there differences in the amounts of time dedicated to use 

for Center-based and District evaluations? 
III. Are there differences in the amounts of time dedicated to use 
for Center-based special education and curriculum programs? 
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FINDINGS 



Table i.O describes allocation of evaluation personnel's tijne for 
the 1980-81 school year and, through January 1982, for the 1981-82 school 
year. Figures are based on weekly time and effort records. Time spent 
on activities related to use, namely proposal development and technical 
assistance is included. Table 1.1 gives the number of hours spent on 
each type of general program evaluation activity. Three activity group- 
ings are Center-Based Special Education, Center-Based Curriculum and 
District programs. 

I. HOW MUCH TIME SHOULD BE SET ASIDE TO PROMOTE USE OF EVALUATION 
INFORMATION? 

Findings presented in Table 1.0 document that about 11% of 
evaluator's time is spent promoting use. Ten percent of an 
evaluator's time was budgeted for use in 1980-81. Findings for 
1981-82 suggest a somewhat higher figure, 11.4%. 
II. ARE THERE DIFFERENCES IN THE AMOUNT OF TIME DEDICATED TO USE 
FOR CRtvnT^R- BASED AND DISTRICr EVALUATIONS? 

The results suggest more time is dedicated to use for Center- 
Based programs. In 1980-81, the percentage of time dedicated 
to use for Center-Based Special Education and Curriculum programs 
was 14.7 and 8.7% respectively. A somewhat lower percentage of 
time was set aside for District accounts (5.9%). Preliminary 
results for 1981-82 document more time being dedicated for use 
at the district level (11.2%). A possible explanation is that 
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district and service center personnel are getting to know each 
other better and are more comfortable discussing evaluative in- 
foi-mat ion. Many district evaluations were initiated the previous 
year . 

If time spent providing technical assistance, a common District 
evaluation activity, is included in use calculations, District 
figures compare favorably with those from Center-Based programs. 
Proposal development, another activity that can be seen related 
to use, is consistently higher for Center-Based programs. This 
correctly reflects that the Center dedicates more resources to 
preparing grant applications. 
III. ARE THERE DIFFERENCES IN THE AMOUN^fS OF TIME DEDICATED TO USE FOR 
CENTER-BASED SPECIAL EDUCATION AND CURRICULUM PROGRAMS? 
More time was dedicated to use for Center -Based Special Education 
programs: 14.7% of evaluator time in 1980-81 and 13.3% for 1981-82 
was dedicated for Special Education programs. The percentage of time 
dedicated to Center-Based Curricular programs was 8.7% and 9.9%. 
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Table 1.0 
How An EvaLuator^s Time Is Used 



PROGRAM TYPE 



Proposal 



1980-81 
Technical 



Using Data 



Development Assistance 



1981-82 
Proposal Technical 
Development Assistance 



Hours 



Hours % 



Hours 



Hours % 



Hours % 



Using Data 



Hours % 



Center-Based 
Special Ed. 

Center-Based 
Curriculum 

District 



119 2.9 



127 
47 



2.1 
1.6 



23 0.6 

43 0.7 
217 7.4 



609 14.7 

530 8.7 
172 5.9 



24 



27 1.2 



99 4.3 



176 13.3 

144 9.9 
259 11.2 



292 2.2 



283 2.1 1311 10.0 



51 1.0 



99 2.0 



578 11.4 



Note: Percentages are calculated using the amount of time available for work. This amount does not include time taken 

for vacations, holidays, sick leave, training, professional development and general meetings. 

a. Based on time and effort records from 14 evaluators 

b. Based on time and effort records from 11 evaluators 
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Table 1.1 
Total Number of Chargeable. Hours 
By Program Type and Year 



Program Type 



Year 



1980-81 



1981-82 
Thru Jan. '82^ 



Cen ter-Based 
Special Education 

Center-Based 
Curriculum 

District 



4148 

6101 
2922 



1319 

1448 
2309 



13172 



5077 



a. 1981-82 are based on time and effort records kept 
through January 1982. 
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NEW DIRF.CTTONS 3 



Knowing an evaJuator spent four hours Tuccting with a reading 
consultant on use of data describes only part of the ongoing process of 
use. Utiat actually happened during the interaction? Did the evaluation 
have an impact? Were objectives made clearer or the curriculum re- 
structured? Or, was the outcome of the meeting sijnply to interpret 
needs assessment findings? 

In November, 1981, evaluation staff began to record one or two 
sentence descriptions of what happened during meetings involving evalua- 
tion use. Excimple conunents include: 

Discussed district assessment. 

Talked about different districts' needs. Will adopt evaluation 
plans to the activities and objectives formulated to address the 
needs . 

Discussed purpose of curriculum mapping and how we plan to use 
data which is presently being collected. Project staff intends 
to use this information for a brief paper as a part of a class. 

Discussed project manager's desire re: data processing for 
questionnaire. Included secretary in meeting. 

Met with project staff to go over evaluation reports on objective 1.0 
materials for TEA on-site visit (monitoring). 

Gave one hour talk to all project part-time staff on evaluation 
findings. Social conversation with full time staff at and after 
lunch. 

Discussed with coordinator survey on services and possibility of 
presenting session on evaluation services at May 6-7 conference. 

A content analysis of comments suggests five kinds of interactions. 
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livnliintion personnel nre currently reviewing the classification 
described below: 

A. Planning evaluation activities (including review of instriunents, 
evaluation plan, etc.) 

B. Reporting evaluation findings (including discussion of possible 
use) 

C. Providing technical consultation (providing input to project 
planning, explaining Levels of Use, assisting with statistics, 
etc.) 

D. Handling logistics of evaluation (obtaining staff lists, 
assembling evaluation plans, etc.) 

H. Socializing with project staff 

Informal discussions during the first part of 1982 have resulted in 
looking at still another dimension. This is the evaluator's perception 
of the interaction. This scale would be completed while the evaluator 
codes time and effort records. The scale under review is: 

I. Project staff demonstrates initiative in the discussion 

for example, they initiate the interaction, propose evalution 
activities, suggest interpretations and uses for evaluation 
findings, etc. 

2. Project staff respond to the discussion in a positive way 
for example, they welcome interaction, show enthusiasm and 
support for the evaluator's proposals, findings, and in- 
terpretations, etc. but they do not initiate ideas themselves. 

3. Project staff respond to the interaction in a neutral way; they 
listen to the evaluator and are receptive to information; however," 
they contribute no significant ideas and little active support. 

4. Project' staff response to the interaction is indifferent; they 
appear to regard the interaction as a mere formality they do 
not contribute to the discussion, and they do not appear to be 
absorbing information presented by the evaluator. 

5. Project staff respond to the interaction in a negative way; 
they are hostile toward the evaluator *s presentation and attempt 
to discredit it. 
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In closing, this research describes progress to date on NIE 
sponsored inquiry. Its purpose is to document how much evaluator time 
is rcc|uired to the use of ov;iI.uation information. Should you wish to 
receiv^e copies of the final report, please write by July 15, 1982 to: 



Alan L. Roecks, Ph.D. 
Coordinator of Evaluation 
Education Service Center, Region 20 
1314 Hines Avenue 
San Antonio, Texas 78208 
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FOOTNOTES 



In the Spring of 1981, per nomination from a national board, the Region 20 
office was visited by personnel from the Harvard-based Huron Institute. 
Personnel studied the office's strong process of using evaluation data. 
The Region 20 office was one of several included in the Huron Institute's 
National research. 



There are two other task codes that account for activities related to use. 
Information from these task codes is presented so the reader can see their 
potential impact. The task codes and their descriptions are: 

Proposal Development • Teclinical Assistance 



Conferences, meetings and discussions. 
Reviewing objectives. 

Preparing evaluation section. 

Reading or reviewing proposals for 
comment . 

Writing complete proposal on 
evaluation topics. 



Travel time connected for 
providing assistance. 

Conducting or participating 
in workshops. 

Other related kinds of 
assistance 



A volunteer group of evaluators have made excellent suggestions on additional 
information that can be captured while recording time and effort records. 
Much of the work discussed in the last section ''New Directions" is a direct 
result of their efforts. The evaluators are Jay Alexander, Tom McGrath and 
Elaine Sebald. 
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Appendix 1: iiSC-ZO Time and HlTort 
Record Keeping System 



This paper describes activities and procedures for ESC- 20 Evaluation 
Services^ time accounting system. With the exception o£ clerical personnel, 
each individual in the evaluation component at the end of the working day- 
records his or her time spent on an activity. This is done through the use 
of the time accounting form which is completed and turned in weekly to the 
person responsible for maintaining the records. Activities are recorded in 
reference to the appropriate prograin, whether chargeable or non-chargeable, 
as well as the task area that effort supports. Printouts by task, project 
hours, employee and project number are obtained monthly (Appendix B) . These 
printouts are basically a management tool for control of the total evaluation 
effort on the project. Totals are maintained and recorded monthly, so that 
the data is readily av^ailablc to know the hourly effort remaining in each 
grant of the program. The system can be replicated by any school district 
that uses multiple funding sources in supporting personnel. Such time and 
effort record keeping systaiis are now required when a person is funded from 
several federal or state funding sources. 
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*lixcerpted from ['lores, R. and Roecks, A. L., Keeping Track of An Evaluator's 
Time: Implications for Better Utilization. Paper presented at the annual 
meeting of the SERA, Dallas, January 1981. 



I'edcral Guidelines Require for Accounting 
of Evaluation Cost 

Cuidclines for evaluating programs suggest evaluation cost be accounted 
for by either of two methods. The first method permits evaluation charges 
to be classified as allocable cost. To use this method, a cost allocation 
plan must be prepared with all costs included in the plan supported by 
fomal accounting records to substantiate all charges. This plan should 
contain: 

a. The nature and extent of services provided and the relevance to 
the federal prograi?.; 

. b. The items of expense; and 
c. The methods to be used in distributing cost. 

If this procedure is used, an arbitrary percentage is not allowed. A per- 
centage can be charged if it is determined by using historical financial 
data based upon the number of man hours required to evaluate a particular 
grant. Different grants would then have different percentages based upon 
the complexity of the evaluation effort, 

The. second method, and the one described in this paper, classifies 
evaluation cost as part of the direct charges involved in the program. 
These costs are accounted for by using time and effort records and charg- 
ing the appropriate grants accordingly. This method requires time and 
attendance records of individual employees, supported by payroll, be used 
to justify the total cost of evaluating the program. If salaries of 
employees are chargeable to more than one grant program, the records must 



- 16 - 19 



show a proportionate distribution of time and effort, Education Division 

(iencr<'jl Administrative liogulation (lilKlAlO states: 

Amoiiiits cluirged to grant prpgrajiis lor persoiuici services, 
regardless of whether treated as direct or indirect cost, will 
be based on payrolls documented cind approved in accordance with 
generally accepted practice of the state or local agency. Pay- 
rolls must be supported by time and attendance or equivalent 
records for individual employees. Salary and wages of employees 
chargeable to more than one grant program or other cost objective 
will be supported by appropriate time distribution records. The 
method used should produce an equitable distribution of time and 
effort. (Federal Register /Vol, 45, No. 66/Thursday, April 3, 
1980/Rules and Regulations.) 

Non-Chargeable Time 

Time spent on activities that cannot be directly related to a funding 
pi'oject are not chargeable and must be accounted for as overhead time. Some 
needs assessment and otli.er data collections are made which support the 
center, but are not reimbursible or chargeable. For non-chargeable activities 
two codes are used: ^^902** which refers to activities associated with opera- 
tion of the evaluation component, and ''020'% which applies to activities 
supporting ESC- 20, in general, but not chargeable to any one project. 
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020 (BSC- 20) 

1. V:ic;iti()n, .^;'u*k, or jxirsoiial nhsencc ;ukI jin'y duly time. 

2. Christnuis cintl lias to i* i use rv ices, training, and rot reals. 

3. Center meetings, conferences, and working on unfunded projects. 

4. Attendance at professional meetings and conferences that are not 
specifically for a given project. 

902 (Component) 

1. Evaluation inservice training and component activity, 

2. Component initiated studies and position papers. 

3. Other administrative activities related to component operation. 

Chargeable Time 

Any time that can be spent on activity directly related to the funded 
program or project is chargeable. Secretarial and clerical personnel, in- 
cluding student help, do not charge against evaluation accounts. Their time 
is considered to be supportive of the overall effort and is included as over 
head. Nine task codes, which break down evaluation activity, are shown on 
the opposite page (Figure 1) . These activities provide for planning of the 
conducted evaluation. Most evaluation time is spent on proposal development 
(task code 2), data collection (task code 4), and interaction with project 
staff (task code 7). Task code 7 was added in the spring of 1980, for the 
purpose of providing additional information on how much time should be set 
aside for working with program staff. Activities describing this new code 
include : 

Contact about developing evaluation plan. 
Briefing, conference, and sign-off of plans, 

.18-21 



Figure 1. Task Couefj 
TASK CODE 

1. NEEDS ASSESSMENT 

* • Consultation with Center personnel desiring data, 

• File search for previously obtained data. 

• Preparation of data collecting Instruments. 

• Selecting sample, collecting data. 

• Tabulating, processing & analyzing results. 

• Report preparation. 

* • Consultation to Interpret & explain findings. 

• Administrative & Supervising. 

CHARGEABLE If related to funded project. 

NON-CHARGEABLE t If a new proposal. 

2. PRO POSAL DEVELOPMENT 

• Conferences, meetings, & discussions. 

• Reviewing objectives. 

• Preparing the evaluation section. 

• Reading, or reviewing proposals for comment. 

• Writing: complete proposal. 

• Administrative & Supervising. 

CHARGEABLE If a continuation proposal. 

NON- CHARGEABLE t If a new proposal. 

3. EVALUATION PLANNING 

• Consultation within evaluation staff. 
« Preparing the Evaluation Plan. 

CHARGEABLE : If related to funded project. 

4. DATA COLLECTION AND PROCESSING 

• Developing or selecting instruments. 

• Travel time connected with collecting data. 

* • Conducting interviews; telephone or in-person. 

. Observations; workshops, schools, centers, etc. 

• Administering exams, tests, questionnaires, etc. 

• Distributing & collecting questionnaires. 

• Briefings pertaining to da^ta collection. 

• Developing a record keeping system. 

• Processing workshop evaluation forms. 

• Tabulating or scoring other instrument data. 

• Preparation for card punching & computer run. 

• Statistical analysis of data. 

• Reviewing records, reports, or other information. 

• Administrative & Supervising 

CHARGEABLE: If related to funded project. 



♦Exclusive of direct interaction with Project Staff. 
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TASK CODE 



5. REPORTING EVALUATION 

• Writing findings, memoranda, interim, or final reports. 
* • Disseminating evaluation results. 

• Administrative & Supervising. 

CHARGEABLE ; If related to funded project. 

6. TECHNICAL ASSISTANCE (LEA) 1 

• Travel time connected with providing assistance. 

• Conducting or participating in V7:;rkshops, 

• Other assistance provided. 

• Administrative & Supervis^ing. 

CHARGEABLE ; If related to a funded project. 

7. INTERACTION WITH PROJECT STAFF 

• Contact about developing evaluation plan, 

• Briefings, conferences, & sign-off of plans. 

• Verbally conveying findings or results of evaluation. 

• Providing technical consultation. 

• Formulating recommendations. 

• Consultations involving using evaluation data. 

• Administrative & Supervising. 

CHARGEABLE ; If related to a funded project. 

8. INTERNAL PROCEDURES 

• All inservice sessions & retreat. 

• Center, component, & unit meetings. 
" Area conferences & workshops. 

• Special studies & projects. ' 

• Preparation of professional papers. 

• Screening applicants for employment. 

• Attendance at professional meetings. 

• Administrative & Supervising. 

CHARGEABLE ; If related to a funded project. 

Prorated for personnel with split fundings 

11. ALL ABSENCES 

' Vacation 

• Sick 

• Personal 

• Jury/military 

• Other 

CT^ARGEABLE; Prorated for personnel with split funding. 



*Exclusive of direct interaction with Project Staff, 
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. Verbally conveying findings or results cf evaluation. 
. Providinv; technical consultation. 
. j'oniiulat i iii; reconimoucla t ion. 

. Consultations involving using evaluation data. 
. Administrative and Supervising. 
Chargeable : If related to a funded project. 

This information was also used for the small scale study discussed later in 
the paper. 

Time Accounting Procedures 

The cost accounting person enters the data provided on the time accounting 
fonius by evaluation personnel into the computer weekly (Appendix B) . The 
amount o£ time recorded each week should correspond to the time for which 
the employee was paid. With part-time personnel, the number of hours for 
which the person was paid must be accounted for. FXill-time staff's hours 
must toteil 40. 

A weekly computer printout is then compared showing the total hours 
available, the hours worked during the period, and the hours remaining in 
the project for evaluation. This printout is basically a management tool 
for control of the total evaluation effort for the project. An exmnple 
printout is included in Appendix B. The computer printout shows the total 
amount of hours worked for each project for billing purposes. The number 
of hours worked during the billing period is multiplied by the hourly rate. 
A special form is filled out and finally transferred from the project to the 
evaluation account. 
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Appendix B 

Printouts by: 
Task, Project Hours, 
Employee, and Project 
Number 





EVALUATION SERVICES 


1 


TINE 


ACCOUNTING REPORT BY TASK 






DECEMBER 19 » 1980 


t 
\ 


TASK 


HOURS 


{ 

I 

i 


1 


157.2 


i 

i 


2 


162.9 




3 


704.5 


Y 


4 


3143.7 




S 


323.8 


I 


6 


267.0 


i 


7 


576.9 




8 


1384.3 


; 


9 


.0 




10 


.0 




11 


346.7 




12 


.0 




TOTAL 


7067.0 


i 
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